Pixel Layering and Layer Propagation for Video Modelling and Foreground Detection

Computer vision applications that work with videos often require that the foreground, region of interest, be clearly segmented from the un-interesting background. To address this problem, we present a general framework for scene modelling and robust foreground detection that works under difficult conditions such as moving camera and dynamic background. This is achieved by first representing the scene as a union of pixel layers, and then propagating these layers through the video by a maximum-likelihood (ML) assignment of pixels to the different layers. The possibility of a pixel not belonging to any of the layers in the scene is also one of the hypotheses that are automatically tested during the maximum-likelihood assignment. The proposed approach has a number of salient virtues. Firstly, the clustering and layering is automatic, while the feature-space can be user defined to suit the application. Secondly, the cluster propagation step implicitly performs layer tracking along with foreground detection. Standard pixel based scene modelling techniques become a particular case of our general framework, when all pixels in the scene are independent and distinct from each other and belong to separate clusters. It is observed that pixels belonging to the same clusters in the feature space usually map to spatially connected layers in the image space, leading us to consider that useful correlation exists between features of pixels in the spatial vicinity. This permits to deal with camera motion with none or nominal registration. We illustrate our ideas with a number of interesting and difficult real-life examples.